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ABSTRACT 

A measure of potential performance as a counselor is 
needed as an adjunct to the information presently employed in 
selection decisions. This article deals with one possible method of 
development of such a potential perf ^-sance criterion and the steps 
taken, to date, in the attempt to val^ .ate it. It includes: the 
overall effectiveness of the instrument relative to the 
discrimination of counselor interpersonal competence; the possible 
modes of presentation of the standardized nroblem-situ&tionc, 
typescript, audio and audiovisual; the relative cost and the relative 
effectiveness of each node of. presentation; and results of the 
convergent-discriminant validation attempt. (Author) 
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Introduction 

The purpose of this research effort was to develop an Instrument 
vhlch could be used as a predictive measure of counseling competence 
based upon performance criteria* 

It is generally conceded that the standard academic criteria 
comroonly employed for graduate school admission serve as poor pre*- 
dlctors for actual counseling performance « Such measures are useful 
predictors with respect to other aspects of the cotmselor's role and 
functions, e.g., the ability to communicate Intelligently, the ability 
to conduct research, etc., but do not Indicate Interpersonal competence. 
Efforts to construct a profile of the successful counselor from the 
usual array of paper and pencil Inventories — Interests, personality, 
values, biographical data, etc., — have been little more rewarding 
(Patterson, 1967 and Whlteley 1968, 1969). 

An empirically derived Instrument which can provide a direct 
measure of counselor competence In realistic performance**type situ- 
ations within the limits of reasonable cost and time expenditure Is 
needed. Consequently, the rationale for the present study was based 
upon a work sample approach utilizing simulated problem**solvlng 
situations* Materials were developed, tested and compared across 
three modes of presentation: typescript, audio and audiovisual. 

In the absence of any established external criteria for evaluating 
counselor competence the rationale depends heavily upon construct 
validity. The present article focuses especially upon the procedural 
aspects of instrument development. 
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Since preliminary results are sufficiently encouraging subsequent 
cross and longitudinal validation In this and other settings are needed. 
With the further development of evUivAce of concurrent and predictive 
validity 9 the Instrument, or a modified version of It i may sei^e as 
an Indicator of Interpersonal potential In a counsellng-type situation. 

PROCEDURES 

Instrument Development 

When potential performance on a complex task must be assessed some 
sample of a subject's present performance on that task can be the best 
Indicator of his potential (Cronbach, 1970). According to Boyd and 
Shlmberg (1971) when such a sampling can lead to Injurious consequences 
for those Involved, It is best to simulate the task. They Indicate 
also that the more precisely the behavior to be a^^essi^d can be specified, 
the better the probability of obtaining a valid measurement. The 
assessment of potential counselor competence, therefore, calls for 
a simulated work sample approach r 

An Instrument to measure the Interpersonal competencies needed 
by a counselor was developed employing the simulated work sample 
concept. The formulation of the competencies desired of a counselor 
Into specific^ measurable behavioral terms; the simulation of a 
counseling Interview to approximate as closely and safely as possible, 
the actual situation to be encountered In the field; and the methods 
by which standardised stimuli could be presented and reliable responses 
obtained In an luexpenslve way were given particular attention In the 
course of lnstraLneT).t. development. 
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Behavior ally defined competencies outlined in the "Ability to 
Belate to Clients" section of the Application to the California Commission 
on Teacher Preparation and Licensing for Approval of Stanford University's 
Schoo l of Education Counseling Psychology Program (1972, p/ 7) were 
employed in a slightly modified form* The modified comptencies were: 

1) Relate to the client in his own language (understanding) 

2) Establish trust and confidence; 

a) Post Hoc Judgment 

b) Does not discuss confidences 

« 

3) Have rapport with client: 

a) Accurate Empathy 

b) Personal CommuAication 

4) Look at client as an individual (do not prejudge client) 

5) Build up client's confidence (in his ability to deal with his 
own problems) 

While good counseling rec-uirea more than just these competencies, 
these seven dimensions tap the interpersonal aspects required. Since no 
adequate indicator of interpersonal potential exists which can be used by 
counselor educators in screening applicants , the instrument was designed 
to produce a measure of each subject's interpersonal competence as 
defined by the above modified competencies. 

In order to simulate a counseling interview as closely as possible, 
actual client problem presentations were solicited from counselors. 
The thirty-five problem presentations, five for each competence, were 
chosen to be representative of the type encountered when working with 
clients ranging from high school age through adulthood. 
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Videotaped interviews were made with actors portraying clients 
presenting the problems to a counselor* These videotape presentations 
simulated counseling situations without the danger inherent in actual 
interaction* They also provided a standard stimulus to which each 
applicant could react. Presentations In typescript form were also 
prepared*. Included were both problem presentations and a bKotI ivon-- 
affective, behavioral description of the client. 

The cost of preparation and ease of administration of the two 
different modes of presentation were noted. The typescript was less 
expensive and easier to administer. Two other modes of presentation, 
audiotape and timed typescript were also prepared for the purpose of 
determining comparative discriminating quality. 

A multiple choice response form was prepared for use with all four 
modes of problem presentation. There were thirty--five sets of responses, 
five corresponding to each of the interpersonal competencies. The 
choices were of four types: a response designed to exemplify the 
competence desired to a high degree; a response which was intended to 
exemplify the competence to a lesser degree; a third response diametri- 
cally opposed to the intent of the competence; and a response which was 
intended to address itself only to the conversational aspects of the 
problem. The responses within each set were then randomly ordered to 
prevent any inadvertent response order biases. 

The typescript problem presentations and corresponding multiple 
choice response sets which comprised the instrument were then administered 
to five behavioral counseling experts from across the country in an 
attempt to validate the response sets. The correctness of response 



was assumed to hold across the different modes of problem presentation. 
Each expert allotted a total of ten points among the four responses In 
each set In accord with his judgment as to the appropriateness of the 
responses to the counseling situation described. Each could allot 
the total ten points to any alternative or any amount to each of the 
four responses as long as only ten points were allotted per problem. 

One of two criteria was employed to judge correctness of response. 
The first criteria was a clear majority, 60 percent, of the points 
allotted by the judges. If the range of judge agreement was between 
fifty and sixty percent and the question was of high difficulty the 
response also was designated correct. In the case of a less distinct 
decision the response or responses were rewritten and resubmitted for 
judgment. Twenty-nine of the questions had response sets which reached 
criterion on the first attempt. The remaining six attained criterion on 
the first iteration. Responses were analyzed to determine Interrater 
reliability using the Hoyt method. A value of .93 was obtained. 

No attempt was made to Induce a time-pressure situation in any 
mode of administration. Administrators of the audiotape and videotape 
modes were Instructed to allow sufficient time for all subjects to 
answer before presenting the next stimulus situation. The timed 
typescript subjects were allowed a total of seventy minutes for re- 
sponding to the thirty-five written situations. Subjects experienced 
no difficulty with the time constraints in any group. The response time 
required in the audio and video groups was from 25-45 seconds per 
question. Most subjects in the timed typescript group completed the task 
within sixty minutes. 
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Construct Validation 

The difficulty In validating an Instrument designed to measure a 
construct Is that no Immediate external criterion exists with which a 
convergent validity Index may be computed. An Indication Is needed 
to determine whether the Instrument Is functioning well enough to 
warrant further Investigation and/or to justify the use of longitudinal 
validation procedures* In order to obtain information on which to base 
such a decision, the "Known Groups" method of construct validation 
(Scott, 1968, p, 253) was employed. The "Known Groups" procedur'^ while 
not providing conclusive evidence does give an indication of the instru- 
ment's worth. The procedure consists of administering the instrument 
to a number of groups of subjects known to differ along a continuum with 
respect to the trait the instrument purports to measure. If the 
instrument does in fact distinguish among the groups in the predicted 
manner there is evidence of construct validity* 

Correlations between total test score and the admissions criteria 
presently employed — years of teaching experience, years of counseling 
experience, undergraduate GPA, graduate GPA, hours of counseling course- 
work, GRE-Qu?xntitative score, GRE-Verbal score, age, sex and marital 
status — were also produced. These were used to form a partial convergent- 
discriminant validity matrix (Cronbach and Meehl, 1955, and Campbell 
and Flske, 1959). This information added to the knowledge both of 
the Instrument and of the present criteria used for admissions. 

Subjects 

Four groups of subjects varying in their ability to relate to 
others were chosen. The first two groups composed of twenty subjects 



each were comprised of professional counselors with at least two years 
work In the field and practlcum students with at least one semester 
of practlcum experience. These subjects represented the high end of 
the ability continuum. The middle was represented by a group of 
twenty-one students from a Foundations of Guidance Course at the 
University of Colorado. This group was similar In composition to 
applicants to counselor training programs. Twenty-one graduate 
physics students constituted the low ability level group because of 
their self-selection into a non-person oriented profession. 

The 82 subjects were randomly assigned to mode of presentation 
within ability levfed grouping. The audiovisual and the untlmed 
typescript groups consisted of twenty-one subjects each while the other 
two modality groups consisted of twenty subjects each. The Instrument 
was then administered, scored, and the results analyzed using a two- 
way Analysis of Variance and Scheffe Multiple Comparison Procedures. 

Cost-Benefit Analysis 

In order to obtain information concerning the differential ef- 
fectiveness of the various modes of stimulus presentation, tables were 
produced based upon the two possible types of Incorrect selection 
decisions. For a number of different cut-off points and several de- 
finitions of those acceptable for counselor training percentages of 
hits, of misses and of false positives were computed for each mode 
of presentation and for all modes combined. 

Rough ratios of cost of presentation of the various modes, 
including preparation, administration, equipment necessary, etc. were 
calculated. This information combined with the effectiveness tables 
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allow an administrator of a counseling program to take into account the 
Importance assigned to each type of selection error with regard to his 
own program as well as the program's size, costs in personnel and costs 
of the various modes of presentation. Discriminating quality of each 
mode of presentation can then be balanced against its relative costs. 

RESULTS 

Construct Validity 

The results of the analysis of variance (ANOVA) of the total test 
scores of all subjects indicated that the instrument possessed signi- 
ficant (p Z-.OOl) discriminating qualities although the various modes 
of presentation did not differ in quality of discrimination. 



Insert Table I here 



Further analysis of the mean differences employing the Sch^ffe 
Multiple Comparisons technique showed that the Instrument discriminated 
among the groups as predicted. 



Insert Table II here 



Insert Table III here 



The graph of the relative distribution of the total test scores for 
each of the ability level groups appears in Figure I. 



Insert Figure I here 
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The only significant correlations found between the supplementary 
data collected on all subjects — years of teaching experience » GPA» 
GRE scores 9 etc.— and >tal test score were those calculated for years 
of counseling experience (.28) > for hours of counseling course work 
(.56), GRE-Quantltatlve Score (-.41) and for sex (.36). These cor- 
relations could be Inflated, particularly the sex- total test score 
correlation, due to composition differences between the high and low 
ability level groups. 

Insert Table IV here 

These findings are consistent with those found by Rank (1966) In a 
study of the relationship between counselor competence and counselor 
perception which euiployed stimulus presentations similar In type to 
those developed for use In the present study. 

Cost-Benefit Analysis 

A rough cost calculation provided a ratio of 4:2:1. Videotape 
mode proved to be twice as costly as audiotape mode and four times 
as expensive as typescript mode* In view of the lack of a significant 
difference in discrimination by various modes of presentation, the 
differential cost incurred by using audiotape or videotape presentation 
does not seem warranted. 

CONCLUSIONS AND RECOMMENDATIONS 
1. The results support the contention that the development of a 
standardized instrument for counselor selection, based upon 
simulated problem situations, is both practical and feasible. 
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2. The discriminating efficiency regarding Interpersonal 
coii5)etence shown by the Instrument, i.e., its ability to 
discriminate among different groups with respect to their 
ability to relate to others , is sufficiently encouraging to 
warrant its further consideration and longitudinal validation. 
Vhile it is true that the evidence provided is not sufficient 
to Indicate conclusive construct validity, the instrument does 
meet the requirements necessary for further investigation into 
its validity. 

3. Contrary to popular expectations which have been generated by 
the promise of more sophisticated technology—videotape 
simulation — the present findings fall to confirm the superiority 
of this medium over simple typescript situations at a level 
sufficient to justify its use in this type of testing sit- 
uation. The authors have observed from the literature that there 
is growing uncertainty regarding the relative advantages and 
disadvantages of the audiovisual medium, possibly due to the 
compounding effects of the multiplicity and types of cues to 
which the subject is exposed. This is a problem, however, 
which goets beyond the scope of the present study. See Dwyer. 
(1970), Fast (1971) and Blrdwhlstell (1970) for further 
elaboration. 

4. No measures of counselor competence exist which have been 
proved both reliable and valid. Development is needed in 
the competence area before either concurrent or predictive 
validity can be assessed accurately. 
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5. While It by no means measures all competencies required of a 
counselor, the results Indicate that the Instrument discussed 
can add a dimension to the selection procedures not at 
present conslierM, I.e., the potential of the candidate to 
develop the Interpersonal skills needed by a counselor. 



NOTE: The Potential Interpersonal Competence Scale (PICS) has been 
copyrighted and an attempt Is being made to keep It secure* 
For further Information concerning the instrument itself or any 
of the tables mentioned but not presented, please feel free to 
contact the authors. 
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TABLE I 



IWO-F ACTOR 4x4 ANALYSIS OF VARIANCE 
MODE OF PRESENTATION BY 
ABILITY LEVEL GROUPING 



Source of 
Variation 


SS 


df 


MS 


F 


Mode 


579.62 


3 


193.21 


1.37 


Level 


11631.10 


3 


3877.03 


27.55* 


Mode X Level 


119A.65 


9 


132.74 


.94 


Residual 


9288.53 


66 


140.74 





* Significant p < .001 



TABLE II 
TABLE OF MEAN DIFFERENCES 



Group Expert Practlcum Applicant Physics 

Means 115.70 (9.9)^ 117.10 (11. 5)^ 101.05 (13. 3)^ 87.33 (12.7)^ 
I 

Expert I 0 -l.A 1A.65* 28.32* 

Practlcum! 0 16.05* 29.72* 

1 

Applicant j 0 13.67* 



* Significant p < .001 

a-standard deviations In parentheses 
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TABLE III 
SCHEFFE TESTS 



Schef fe Comparlsona 



Lparlson 


.^PE 




w 


Sph 




F 




1 


-1 


0 


0 


.14 


F(1.38) 




0 


0 


1 


-1 


13.94 


F(l,40)* 




1 


0 


-1 


0 


15.62 


F(l,39)* 




1 


0 


0 


-1 


64.29 


F(l,39)* 


*5 


0 


1 


0 


-1 


58.38 


F(l,39)* 




0 


1 


-1 


0 


18.75 


F(l,39)* 


^7 


h 


h 


-i 


0 


U.62 


F(2,58)* 


i/»8 


h 


h 


0 


-1 


41.54 


F(2,58)* 


i/,9 


h 


h 






23.88 


F(3,78)* 



MS^, - 140.73535 

* Significant p < .001 



TABLE IV 

CORRELATIONS WITH TOTAL TEST SCORE 



Variable 


Correlation 


Mr in 


S.D. 


Levftl Value 


-.67** 


2.52 


1.12 


Quart lie 


.94** 


2.50 


1.11 


Years of Teaching Experience 


.13 


3.18 


3.49 


Years of Counseling Experience 


.28* 


2.15 


3.54 


Undergraduate CPA 


-.10 


3.08 


.40 


Graduate CPA 


.21 


3.65 


.25 


Hours of Counseling Course Work 


.56** 


15.70 


19.90 


GRE-Quantltatlve 


-.41** 


595.54 


134.02 


GRE Verbal 


.05 


591.69 


114.47 


Age 


.20 


30.06 


6.05 


Sex (Female-l, Male-O) 


.36** 


.38 


.49 


Marital Status (Marrled-2, 

Single-1) 








.14 


1.59 


.84 



* significant p -'^.05 



** Significant p < .001 
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Figure I 

Distribution of Test Scores by Group 



